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ABSTRACT 

This module underscores the need for evaluation in 
order to collect baseline data to monitor progress and to make 
decisions about needed changes in rural school career guidance 
programs. Evaluation is a process that occurs during planning and is 
used to monitor progress, and, later, to assess overall program 
impact. After a delineation of commonly held negative perceptions of 
evaluation, the definition of evaluation and the nature of the 
evaluation ^beast^ are discussed. With extensive audiance 
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considered in evaluating the planning phase of program 
development/implementation is developed. Using two handouts included 
in the module, participants are instructed to work in small groups to 
develop a similar matrix for evaluating the planning of a staff 
development program. Evaluation concerns explored include issues such 
as: who are the users of evaluative information and what are their 
information needs; strategies for collecting and reporting that 
information; phases of prfpgrams (planning, tryout, implementation), 
and sample evaluation considerations for planning and implementation 
phases. Nximerous evaluation instruments and references for others are 
provided. (NEC) 
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FOREWORD 



The National Center for Research in Vocational Education has been actively developing mate- 
rials for the special guidance program needs of rural and small schools since 1975. During the en- 
duing period, a 16-volume set of career guidance program materials for rural and small schools 
entitled the Rural America Series viss produced. The development and original production were 
funded by the U.S. Office of Education, Bureau of Occupational and Adult Education, Part "C," 
Vocational Education Act of 1963 as amended Researc^ Legislation. Since its completion, the 
Rural America Series has enjoyed widespread national distribution with over 1 ,200 sets now in the 
field. In 1976-1978 the National Center in conjuncxion with Northern Michigan University was 
funded to test the planning prcx;ess contained in the Series and to develop several supplemental 
materials. 

One such supplement is the Facilitator's Guir/e to Staff Training for the Rural America Series. 
It consists of 18 separate modules of which ihis document is one. The Guide is designed to assist 
state department of education personnel, guidance supervisors, and guidance counselors in training 
others to jnderstand and utilize the Rural America Series. Each module of the Guide may be used 
independently or collectively in staff development activities. All parts of the Guide have been 
thoroughly tested in state-wide workshops sponsored by the states of Florida, West Virginia, Ohio, 
Illinois, Iowa, Arkansas, Kansas, Washington, Idaho, Wyoming, and the territory of American 
Samoa. Each workshop was carefully evaluated and suggestions from workshop participants were 
used to modify and improve the Guide. 

"Evaluation" underscores the need for evaluation in order to collect baseline data to monitor 
progress and to maKe decisions about needed changes in the program. Numerous evaluation instru- 
ments and references for others are provided. Training in viable evaluation techniques in concert 
with the other parts of the Guide should prove to be valuable assets for the staff development of 
rural and small school counselors and educators as they plan, implement, and evaluate career guid- 
ance programs for their students. 

Robert E. Taylor 
Executive Director 
The National Center for Research 
in Vocational Education 
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MODULE XII: EVAL UATION 



Module Overview 



Instructional Time 

Approximately 2 hours 



Module Description 



The Rural America Series is initially presented with the notation that evaluation is a 
process that occurs both during planning, to monitor progress, as well as later on to assess 
overall program impact. After a delineation of commonly held negative perceptions of evalua- 
tion, the definition of evaluation and the nature of the evaluation "beast" are discussed. 
With extensive audience participation, a matrix of the major evaluation elements to be con- 
sidered in evaluating the planning phase of program development/implementation is 
developed. Following the presentation, the facilitator explains two handouts-Major Evalua- 
tion Elements for Program Planning, and Major Evaluation Elements for Program Implementa- 
tion. The participants work in small groups to develop a similar matrix for evaluating the 
planning of a staff development program. A brief summary concludes the module. 



Goals and Objectives 
Participants will 

Goal 1 Gain an understanding of the evaluation process. 

Objective 1.1 Identify the differences between formative (program planning) evalua- 
tion and impact (prognm implementation) evaluation. 

Objective 1.2 Develop preliminary evaluation plans for both the planning of a staff 
development program and the impact of the program on school staff. 



Agenda 

5 minutes 
10 minutes 

45 minutes 

45 minutes 

5 minutes 



Introduction 

Self-Scored Quiz 

The Rural America Series 

Evaluation— Concepts and 
Ideas 

Developing Preliminary 
Evaluation Plans 

Module Summary 



Presentation 

Individual Activity 
Presentation 

Presentation and 
Brainstorming 

Small Group Activity 



Presentation 



XII-5 
XII-11 
Xli-23 
XI 1-27 



XII-1 



Module Evaluation 

To measure the success of the module, collect and analyze the completed matrices 
developed by the small group. 

If the module is used independently of other modules in this set then the module- 
specific questionnaire in the participant materials may be administered. If the module is 
part of an extended (multiple-day) workshop, the use of the overall workshop question- 
naire (see Appendix A) is recommended. 



Relationship to the Rural America Series 
See chart on the next page. 
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THE RURAL AMERICA SERIES (HANDBOOKS) 



State of the Art 



Model 



Career Guidance 
Resources 



PROBLEM IDENTIFICATION 
Support PIsnning 
Information Process 



Planning and 
Implementation 



Needs 
Assessment 



Behavioral 
Objectives 



Resource 
Assessment 



Evaluation 
(Module XII) 



Career Guidance & 
Counseling Strategies 



PROBLEM RESOLUTION 



Career Counseling 



Desk Reference 



Individualized 
Counseling and 
Placement 



Transitional 
Career Placement 



Career Guidance 
Practices 



Support 
Functions 



Staff 

Development 



Community 
Relations and 
Involvement 



Community 
Perspectives 



Supplemental 
Areas 



Facilitator's 
Guide 



Cooperative 
Agreements 



Case Studies 



Evaluation is an essential element of the planning process. Although evaluation pervades all activities of a comprehensive 
career guidance and counsehng program, it is important to emphasize that the planning process itself must also be evaluated. A 
program that is carefully evaluated during its planning phases will more likely be successful when it is fully implemented. The 
evaluation handbook of the Rura/ America Series contains evaluation procedures for both planning and implementation phases 
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ACTIVITY TITLE: DURATiOiSi: 
Self Scored Quiz and the 

Rural America Series 1 0 mi nu tes 



FACILITATOR OUTLINE 



A, The Quiz 



NOTE: Your explanation to the participants is really 
"tongue-in-cheek." The quiz, while containing a few grains 
of truth is primarily motivational in nature. 



1. Explain that it is important to obtain some preliminary 
information or baseline data before fully starting into 
the content of this module. 

2. To that end, this module contains a brief quiz designed 
to discover understandings and perceptions of evaluation 
concepts. 

3. The quiz could be used as a pre- and post- test but we will 
only use it as a self-scored pre-test. 

4. Stress the importance of the quiz and that it is necessary 
for everyone to work independently and quietly. 



Pass out the quiz (participant handout, p. XI 1-29) to partici- 



pants and give them a few minutes to complete 



5. Give participants the correct answers and ask them to 
score their own quizzes. The key to the quiz is given 
below. 



NOTES 




Are they putting me on? 



EVALUATION QUIZ 

FOR THE RURAL AMERICA SERIES 

Directions; Each of the following questions was selected to measure 
your understanding of evaluation. Read each question carefully and 
then circle the letter of the statement that best represents your 
answer to the question. After you have completed the quiz, the 
module leader will give you further instructions. 
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FACILITATOR OUTLINE 



NOTES 



Quiz Key (Continued) 



Evaluation Multiple Choice Quiz 

(Keyed Quiz for Facilitator) 



1. Which of the following words or phrases best describes your 
first thought when you hear the word evaluation? 

* a. Hooray. 

b. O.K., already. 

c. Ugh! 

d. Yuck! 

2. How would you describe evaluation? 

a. Testfng program. 

b. Something which I'll probably have to do. 

c. Someone rating or ranki:ig you. 

d. All of the above. 

* e. A valuable program and p anning tool. 

3. How would you characterize your abilities to conduct a 
program evaluation? 

a. It's difficult for me :o do. 

b. I need a consultant. 

c. You've got to be kidding. 

^ d. Sometimes easy/sometimes difficult. 

4. In your judgment, which animal, or beast, represents 
evaluation. 

a. Teddy bear (warm and cuddly) 

* b. Kitty cat (fun but at times hard to understand) 

c. Fully scented skunk (self-explanatory) 

d. Ogre (self-explanatory) 



6. By item, ask how many participants got each question 
correct. Tally the responses on the chalkboard as shown 
below. (Hypothetical data supplied.) 



« « « # 



Item 



No. Correct 



No. Incorrect 



1 
2 
3 
4 



3 
10 
12 

8 



27 
20 
18 
22 
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FACILITATOR OUTLINE 



NOTES 



B. Quiz Interpretation 

1. For the most part the answers (unless the participants 
have "psyched" out the obvious) will load hepvily on 

the negative interpretations and perceptions of evaluation. 

2. Point out that evaluation is often perceived in a negative 
light. Common perceptions are: 

• Testing programs. 

• Something deemed as important by administrators 
and individuals high up in the system. 

• Someone rating or ranking you. 

• Something that really doesn't have much value. 

3. As with most stereotypes there may be some truth to 
these perceptions but they are narrow and very limiting. 

4. Evaluation is really part of our everyday lives. Examples 
are: 

• Did I improve my Jogging time or distance today? 

• How good does th^s wine taste in comparison to 
others? 

• Is my garden growing according to schedule? 

• Which of these two cars gives the best economy and 
performance? 

As an option, you might want to re^i the following 
excerpt which shows in a humorous way how evaluation 
pervades our everyday life. The excerpt is taken from 
"The Jogging Fad: A Non-Jock's Lament," by James B. 
Kobak, Jr., WA Ambassador, p. 18, October, 1977. 

"Running slowly for moderate distances may be done 
almost anywhere, and is best engaged in when the lawn 
looks as if it might need mowing or one's spouse looks 
as if she would like the sunporch painted. Running 
slowly for moderate distances was once considered 
sprinting rapidly over vast stretches of mountainous 
terrain. Three events caused this estimate to be revised: 
first, the author's wife purchased a device that 
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FACILITATOR OUTLINE 



NOTES 



accurately recorded the number of miles run; second, 
the author's wife purchased a device that accurately 
recorded the passage of time; and third, the author 
was passed on a small hill by someone else's wife in 
much the same manner that the Washington Monument 
is passed by a 707. Someone else's wife proceeded to 
sprint up the hill not only as rapidly as Bambi's mother, 
but also as effortlessly." 

The evaluation implications in this short quote, 
especially in points one-three are obvious. 

5. Evaluation is a tool that we use in making decisions. 
It is a part of the decision-ma^jr.g process. 

6. In this session, we will be taking a closer look at 
evaluation. 

• The role of evaluation in the Rural America Series, 

• The evaluation process. 

7. Hopefully we will be able to dispel many of the stereo- 
types regarding evaluation. 

C. Evaluation and the Rural America Series. 

1. Evaluation is frequently a process or a step tacked onto 
the end of a diagram or flow chart. 

2. You sort of do it as ar afterthought to an event or 
series of events. 



Show the transparency, "Rural America Series," p. XI 1-36. 



3. Using the transparency, emphasize that there are numer- 
ous steps and procedures involved in planning and imple- 
menting career guidance programs. These are: 

• Developing a theoretical base for the program {State 
of the Art and The Model boxes on the flow chart). 

• Planning for the program including Needs Assessment, 
Behavioral Objectives, Resource Assessment and the 
utilization of existing resources (Career Guidance 
Resources and the Desk Reference). 
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FACILITATOR OUTLINE 



NOTES 



• Selecting and implementing Career Guidance 
Practices, Career Counseling programs and Transitional 
Career Placement programs. 

• Implementing meaningful Staff Development programs, 
assessing Community Perspectives and providing for 

m janingful Community Relations and Involvement 

• And finally— the evaluation of the planning and imple- 
mentation phases of the program. 

4. The Series is a rich set of resources for comprehensive 
career guidance programs in rur?J schools. 

5. Evaluation is placed at two different locations in the 
Series. 

6. Evaluation in the planning phase emphasizes: 

• That the planning process itself must be evaluated. 

• That the planning of evaluation must be started early 
and not tacked on at the end of a program. 

7. Evaluation at the end serves to illustrate the need for 
overall program evaluation. Evaluation that: 

• Systematically assesses program outcomes. 

• Helps decision makers. 
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ACTIVITY TITLE: 



DURATION: 



Evaluation—Concepts and Ideas 



45 minutes 



FACILITATOR OUTLINE 



NOTES 



A. Evaluation Defined 

1. Evaluation Is the process of providing information for 
making judgments about decision options. In less formal 
language, evaluation provides information for making 
decisions. 

2. Inherent in the process described in the definition are 



such questions as: 

• Who needs the information? 

• What information do they need? i.e.. What questions 
do they need answered? 

• How will that information be collected or obtained? 

• How will that information be reported? 



B. Operationallzing the Definition 

1. To make the definition meaningful (and workable) let's 
take a look at what might be happening in a typical school 
district during the first year of implementation of the 
Rural America Series. 



Ask the participants to help you by brainstorming what might 
b3 happening in this typical district. Write their responses on 
the chalkboard. 



2. Activities might include: 



• Organizing a planning committee. 

• Selecting instruments for a needs assessment. 

• Conducting a needs assessment. 

• Writing behavioral objectives. 

• Assessing local resources. 

• Determining staff needs. 
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FACILITATOR OUTLINE 


NOTES 


• Planning staff development programs. 




• Planning home/school/community career guidance 
programs. 




• Working out cooperative arrangements with business, 
industry and labor. 




• Developing placement programs or improving existing 
ones. 




• Assessing community attitudes. 




• Evaluating parts of programs or the planning process 
itself. 




3. Summarize the list of activities. 




• The activities in the first nine months (school year) of 
the project are primarily focused on program planning 
rather than program implementation. 




• It takes effort and time to plan a solid, strong career 
guidance program. 




• Only small programs or parts of programs will be 
implemented in the first year. 




4. Describe the three basic phases of program development 
and implementation. 


pp. 4-7, Deciding Via 
Evaluation 




Write the three phases with descriptions on the chalkboard. 
Explain the three phases as shown below. 






PHASE 1 - FIRST YEAR, PROGRAM PLANNING 




• Heavy emphasis on program planning. 




• Limited tryout of new programs or parts of programs. 




• Very limited (and possibly no) full scale program 
implementation. 




PHASE 2 - SECOND YEAR, PLANNING AND 
TRYOUT 




• Program planning still continues but is not as heavily 
emphasized as in Phase 1. 
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FACILITATOR OUTLINE 



NOTES 



• Heavy emphasis on the tryout of new programs or 
expanded ones planned in Phase 1. 

• Increased program implementation. 

PHASE 3 - THIRD YEAR AND BEYOND, 
IMPLEMENTATION 

• Program planning is a minor but continuing emphasis. 

• Limited tryout of new programs. 

• Full-scale program implementation/impacting on stu- 
dents. 

C. Evaluation of the Planning Process— General Concepts. 

1. Evaluation of Phase 1 (Note: The evaluation handbook 
generally emphasizes the evaluation of the program plan- 
ning process). 

• The earlier discussion provided a framework or back- 
drop against which an evaluation peispective .;an be 
developed. 

• Recall that the definition of evaluation suggested four 
major questions: 

— Who needs the information? 

— What kinds of questions do they need answered? 

— How can that information be collected/obtained? 

— How should it be reported? 

2. Develop a matrix for the evaluation of program planning. 



This activity uses a brainstorming technique in which the 
facilitator asks questions that enable participants to offer 
suggestions for completing the matrix. If the participants 
slowdown in terms of suggestions, supply answers so that they 
will get a sense or feel for evaluative thinking. A sample com- 
pleted matrix is enclosed in the facilitator outline. The ques- 
tion cues and their comments are found in the facilitator 
outline directly after the completed matrix. 

Place the matrix framework headings on the chalkboard as the 
brainstorming session starts. Explain the column headings as 
you place them on the board. Provide participants with blank 
matrix copies for their use in note taking. See page XI 1-30 in 
the participant materials. 

Note: Each matrix column was developed in a vertical fashion. 
Therefore, there is not a one-to-one correspondence across the 
columns. 
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EVALUATION OF PHASE 1-PROGRAM PLANNING 
(Completed Sample Matrix for Facilitator Use) 



^\Evaluatlon 
Level ^"""^^^---^ 


Who Is (Are) 
the Decision 
Makers? 


What Questions or 
Concerns Do 
They Have? 


How Can That 
Information Be 
Collected/Obtained? 


How Should That 
Information Be 
Reported? 


ADMINISTRATIVE 


• Superintendent 

• School board 

• Local community groups 

• School administrators 
(principals, department 
leaders, etc.) 

• State officials (especially 
if project is state sup- 
ported) 

• Indirectly involved 
school staff 


Overall Administrative 
Questions: 

• Who is involved and who 
might be involved m the 
program? 

• What is the program in- 
tended to accomplish and 
what is it actually accom- 
plishing? 

• Is the program on schedule? 

• Are there changes neces- 
sary in the program- 

— Program expansion/ 
reduction? 

—Personnel shifts? 

-Problems being encoun- 
tered? 

• How much is the program 
costing? 

• Will additional monies be 
required? 


(See Below) 


• By reports to the Board on 
a monthly or less fi'equent 
basis. 

• By short reports on a 
monthly basis to key ad- 
ministrators. 

• By t 'e of short 5-page or 
preferably less, memoran- 
dums to ^he decision 
makers. 

• By product displays. 

^ By the use of brief sum- 

ITloiriZcQ uala IdUlca. 

(If possible some of the 
above memos or tables 
might be used as the basis 
for local newspaper articles) 


PROGRAM 


• Program coordinator 

• Directly involved school 
staff and administrators 

• Overall planning com- 
mittee and subcommittees 

• Involved students 

• Other involved individuals 
(e.g., community members) 


Frequent Project Level 
Questions: 

• Is the program on 
schedule? 

• Are the products of our 
work meeting a high 
enough standard? 

• What are the problems that 
are being encountered? 
(This question is especially 
important if programs or 
parts of programs are 
being tried out.) 

• What's happening in other 
parts of the project or in 
othe- subcommittees that 
I'm not involved in? 

• If new programs a:e being 
implemented, to what ex- 
tent are they having an 
impact on students, teach- 
ers, etc.? 


• By carefully developing 
program objectives that 
specify end products and 
completion dates. 

• By closely monitoring such 
program facets as 
-Schedule 

-Costs 

-Product Quality 

• By keeping informed of 
program effects (e.g., call- 
ing committee chairpersons, 
attending committee meet- 

tnns otr ) 

• If new programs are im- 
plemented, bycondtcting 
preliminary assessments of 
impact. 

• By using results of student, 
community, etc. surveys 
that are a part of first year 
activities? 


• By use of short, f.equent 
reports to the overall plan- 
ning committee. 

• By calling individuals with 
reports of progress. 

• By sending short summar- 
ized reports to subcom- 
mittees. 

• By having frequent plan- 
ning committee meetings 
(approximately every 2-3 
weeki) where the above 
reports are an agenda item. 
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FACILITATOR OUTLINE 



NOTES 



• Facilitator Questions for Matrix Column One 

- Who are the individuals or groups who might be 
making decisions about the program? 

- Are there other decision makers (besides the 
obvious school board, superintendent, etc.)? 

• Comments for Matrix Column One 

- There are two distinct levels of decision making— 
the administrative and the program level; the ques- 
tions they need answered and the frequency with 
which they need information differ somewhat Ad- 
ministrative level refers to those individuals or 
groups who make decisions about the program but 
are not involved in it on a day-to-day basis. Program 
level refers to those individuals or groups directly 
involved in the program. 

- One group is at a higher level with less frequent 
information needs. The other requires a much 
speedier information flow if the planning process 
is to succeed. 

• Facilitator Questions for Matrix Column Two 

- What questions or concerns might the administra- 
tive level have during the program planning phase? 
(Note: Some questions that participants suggest 
might be more philosophical than evaluative in 
nature. For example: "Should our district be in- 
volved in this program?" For this type question 
mention that it is more philosophical in nature 
and then quickly return the group focus to evalua- 
tive questions.) 

- Are there other different questions that individuals 
and groups on the program level might have? 

• Comments on Matrix Column Two 

- Generally both sets of decision makers have similar 
concerns. 

- However, the program level decision makers need 
information on a day-to-day basis. They need rapid 
evaluative feedback of a more specific nature. 

- The administrative concerns are at a higher level 
and need to be responded to on a much less frequent 
basis. 

• Questions on Matrix Column Three 

- How might some of the information that decision 
makers need be collected? 
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FACILITATOR OUTLINE 



Cue the participants by asking them to think back to earlier 
workshop concepts such as program and behavioral objectives, 
student assessment, etc. 



- iVIight not the same general procedure for informa* 
tion collection be used at both levels of decision 
making? 

• Comments on Matrix Column Three 

- For the most part, the same procedures will work 
for both levels of decision making. 

- The data collection procedures in the first year rely 
heavily on the degree to which the project coordina- 
tor monitors the progress of the project-i.e., the 
degree to which the coordinator keeps adequate logs 
and records of project progress. 

- Careful project planning and the development of 
program objectives, schedules, cost figures and 
responsibilities are the essential starting point for 
collecting information in the planning year. 

- Tangible products (e.g., completed needs assessment 
documents, lists of behavioral objectives, resource 
catalogs) are also supportive of events and activities 
taking place and reaching successful fruition. 

• Questions and Comments for Matrix Column Four 

- Here both the questions and comments are fairly 
self-evident. In fact, this column could be presented 
in almost a lecture fashion to participants by the 
facilitator. However, it would still be wise to stress 
several points. 

- First, for both the administrative and program levels 
keep all reports short (5 pages or less) and to the 
point. 

- Second, report at the program level considerably 
more often than at the administrative level. 

- Lastly, capitalize on the information collected by 
getting articles in local newspapers, reports to 
parents, etc. 

3. Summarize the evaluation of the planning phase. 

• Note that the purpose of the previous activity was to 
develop understanding of how the planning phase of 
the program could be evaluated and how one could go 
systematically through the thought processes associated 
with evaluation. 
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FACILITATOR OUTLINE 



NOTES 



• Stress that consultants can and often are employed 
to help with the evaluation process, but that the proj- 
ect coordinator and program level personnel should be 
heavily involved in both the conceptualization and the 
operationalization of evaluation procedures. 

D. Evaluation of the Implementation Phase-General Concepts 



Erase the completed cells of the previous matrix except for 
the first and last coumns. Also,erase the matrix heading and 
replace it with Phase 3-Program Implementation. 

Note: Some minor changes for this phase have also been made 
in the last column. As the facilitator you might want to 
quickly place these changes on the fourth column. 



1 . Explain that for a brief period of timf. we will examine 
the concepts involved in thb evaluation of program im- 
plementation-especial ly full scale program implementa- 
tion. 

• The same basic kind of matrix, as just completed, can 
be used to look at the evaluation of full scale program 
ir.^plementation. 

• We can use the same headings and questions as before. 

• The decision makers will also be the same as before. 

2. Develop the rest of the Matrix for the Evaluation of 
Program Implementation. 



The facilitator should complete the matrix in lecture fashion 
with a small amount of participant involvement. 

A sample completed matrix is enclosed in the facilitator 
outline. 



• Comments on Matrix Column Two 

- Generally speaking, both the administrative and 
program levels have similar questions. 

- Both need to know the kind of impact the program 
is having on the wide variety of individuals associ- 
ated with it. 

- The questions here relate to the kinds of decisions 
that each level has to make. 

* Administrative— cost decisions, continuation/ 
termination decisions 

* Program— program improvement decisions, in- 
course corrections. 
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EVALUATION OF PHASE 3-PROGRAM IMPLEMENTATION 
(Completed Sample Matrix for Facilitator Use) 



Evaluation 
Concern 



Level 



Who Is (Are) 
the Decision 
Makers? 



What Questions or 
Concerns Do 
They Have? 



How Can That 
Information Be 
Collected/Obtained? 



How Should That 
Information Be 
Reported? 



ADMINISTRATIVE 



PROGRAM 



• Superintendent 

• School board 

* Local community groups 

* School administrators 
(principals, department 
heads, etc.) 

* State officials (especially 
If project is state sup- 
ported) 

• Indirectly involved school 
staff 



Overall, how well is the 
program doing? 
What kind of changes or 
demonstrable impact is 
the program having on; 
Students? Teachers? Com- 
munity? Administrators? 
Are there parts of the pro- 
gram that should be ex* 
panded? deloted? 
What new Or unusual costs 
is the program requiring? 
How many students are 
being reached? 
Have student self under- 
standings, career aware- 
ness, etc. changed? 
How satisfied are partici- 
pants with the program? 
What are the long term, 
follow-up results of the 
program? 



(See Below) 



By reports to the Board on 
an interim basis 
By short reports on a 
monthly basis to key ad- 
ministrators 

By the use of short 5*page 
or preferably less, mem- 
orandums to the decision 
makers. 

By product displays 
By the use of brief sum- 
marized data tabids (If 
possible some of the above 
memos or tables might be 
used as the basis for local 
newspaper articles.) 



• Program coordinator 

• Directly involved school 
staff and administrators 

• Overall planning com- 
mittee and subcommittees 

• Involved students 

• Other involved individuals 
(e.g., community members) 



* Overall, how well is the 
program doing? 

* What kind of impact are 
we having? 

—Impact on Students 
-Changes in Teacher 

Behavior 
-Community Invo'vement 
-Administrative Support 

* Have student self under* 
standings, career aware- 
ness, etc. changed? 

* How satisfied are partici- 
pants with the program? 

* Are there activities and 
procedures that we can 
change to make the pro- 
gram work better? 



Continued careful moni* 
toriny of the program in 
terms of costs, schedules, 
responsibilities, etc. 
Developing and implement- 
ing a program evaluation 
plan that might include 
such factors as: 
—testing programs 
—longitudinal assessments 
— pre*posttest designs 
-control group designs 
—sampling procedures 
—statistical designs 
—normative designs using 

already existing data 
-follow-up studies 
—parent/community 

questionnaires 
—staff surveys 



By use of short, frequent 
reports to the overall 
planning committee. 
By calling individuals with 
reports of progresses. 
By sending short summar- 
ized reports to subcom- 
mittees. 
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FACILITATOR OUTLINE 



NOTES 



— Lastly, the kinds of questions posed are really the 
bottom line with regard to the program, i.e., did . 
it really have an impact? 

• Comments on Matrix Column Three 

— Stress that continued program monitoring tech- 
niques are still very important. For example, the 
number of students involved in the program, the 
cos^s, tho amount of time spent in inservice training, 
etc. are easily obtained via simple project monitor- 
ing strategies. 

— Data collection techniques as indicated on the 
matrix convey some of the connotations of the 
term evaluation that were discussed earlier. 

* One might now think of such things as testing 
programs that measure changes in student aware- 
ness, career maturation, career understandings, 
etc. over time. These testing programs could be 

at grades 6, 8, 10, and 12, providing a longitudinal 
picture of program impact. This type of evalua- 
tion is underscored if state or national funds 
were being used for project activities. 

* Other factors such as pretesting and posttesting, 
the use of control groups, statistical analyses, etc. 
are all important considerations at this point. 

* The types of variables to be assessed are also very 
important. Behavioral objectives as described 
earlier play a key role in determining what is to 
be assessed. The objectives must be carefully 
reviewed before assessment strategies are selected. 

* Parental, community, staff and even student sur- 
veys might be used periodically to assess percep- 
tions of the program and satisfaction with it. 

* Standardized instruments will most likely be used 
for the evaluation of the implementation phase. 
A sample of instruments might include: 

Career Maturity Index— John 0. Crites, Career 
Maturity Inventory (CTB/McGraw-Hill, Del 
Monte Research Park, Monterey, Calif., 1973). 

Strong-Campbell Interest Inventory— Edward K. 
Strong, Jr. and David P. Campbell, 5fro^flf- 
Campbell Interest Inventory (Merged Form), 
(Stanford, California: Stanford University Press, 
1974). 

Occupational Interest Survey-Frederic G. Kuder, 
Occupational Interest Survey (Chicago: Science 
Research Associates, 1966). 




Did the program really 
have an impact? 
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FACILITATOR OUTLINE 



NOTES 



Career Choice Inventory- Robert E. Ripley, 
Career-Choice Inventory (Mesa, Arizona: Johnson 
Publishing Company, 1973). 

Career Assessment Inventory-Charles B. 
Johanson, "Career Assessment Inventory/' The 
National Computer Systems, Inc., Minnesota, 
197b. 

Texas Career Education Measurement Series- 
Westinghouse Learning Corporation, 'Texas 
Career Education Measurement Series," produced 
foi the Texas Education Agency and the Partners 
in Career Education Project, 1975. 

Career Awareness Inventory-Laverna M. Fadale, 
Career Awareness Inventory (Ithaca, New York: 
Cornell Institute for Research and Development 
in Occupational Education, undated). 

- Additional instruments can be located in the Series 
handbook entitled Career Guidance Resources and 
in a set of materials dealing with career education 
measurement. (Norval L McCaslin, Career Educa- 
tinn Measurement Handbooks, The National Center 
for Research in Vocational Education, The Ohio 
State University, in process.) 

- Generally speaking, consultant help with the 
development and implementation of an evaluation 
plan for program innplemantation is required. How- 
ever, it is crucial that the program coordinators 
(and involved staff) should take an active leader- 
ship role in stating evaluation questions and in 
defining what Is to be evaluated. 

• Comments on Column Four 

- As is obvious the reporting of information here is 
similar to that for Phase I, Program Planning. Note: 
Evaluation results for Phase 3 are reported less fre- 
quently than for Phase 1 . 

- The same basic procedures can be utilized for report- 
ing in Phase 3 as were utilized in Phase 1 . Since out- 
comes in Phase 3 relate more directly to the impact 
of the program on students it is highly desirable to 
get project results published in local newspapers. 

- The results should be used to examine strengths and 
weaknesses in the program and to reshape or modify 
those parts of the program that do not work well. 
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FACILITATOR OUTLINE 



NOTES 



• Additional Comments 

- It should be noted that matrices for only two of the 
three phases (1 and 3) have been developed. Ob- 
viously the matrix for Phase 2— Program Tryout 
would be a combination of ideas from Phases 1 and 
3, with respective emphases dependent upon 
whether the project was just entering the tryout 
phase or was nearing completion of that phase 

and ready to start full scale implementation. 

- Across all three phases of program development 
(planning, tryout and implementation) the basic 
process of evaluative thinking will be similar. Tech- 
niques will vary from phase to phase (e.g., plan- 
ning utilises project monitoring techniques whereas 
as the approach for implementation includes such 
strategies ai, experimental design, longitudinal test- 
ing and so forth) but the logical process is the same. 
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ACTIVITY TITLE: 

Developing Preliminary Evaluation Plans 



DURATION: 

45 minutes 



FACILITATOR OUTLINE 



NOTES 



A. Orientation to the Small Group Activity 



1. The purpose of the activity is to provide "hands-on" 
practice in thinking like an evaluator. 

2. The activity is open-ended and requires that partici- 
pants work in small groups to brainstorm ideas for the 
evaluation of the planning and implementation of an 
inservice career guidance program for a local school 
district. The program might include: 

• Monthly after-school training sessions. 

• Extensive use of the Rural America Series and specific 
handbooks in the series. 



• Simulations, games, small and large group activities, 
lectures, and so forth. 

3. The activity is designed for participants to use their 
imagination in thinking through a staff development 
program for a district. Capitalize upon the skill, back- 
ground and expertise of those individuals ir .he small 
groups. 

B. Specific Small Group Instructions 



Distribute the handout entitled "Evaluation of Phase 
p. XII-30. Explain the steps listed below. 



1. Appoint a leader for each small group. Assign half 

of the groups the evaluation of the planning phase-the 
other half the evaluation of the full-scale implementation 
phase. 

2. Each group should spend about 10 minutes or so brain- 
storming the planning for or implementation of an in- 
service program in career guidance for school staff 
members. 

Distribute the handout entitled, "Staff Development Ideas," 
p. XI 1-31. Note that the handout is a set of idea generators 
for this activity. 

Suggest to participants that they be creative in their staff 
development program. 
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FACILITATOR OUTLINE 



NOTES 



3. Up to this point the product of this activity should be 
a brief outline for a staff development program. 

4. As a small group, complete in brief note form matrices 
for the evaluation of the planning and implementation 
phases of the inservice program, whichever is appropriate. 

5. Some question guidelines to help generate matrices 
follow: 

PLANNING PHASE 

• How much will this training cost in terms of time, 
resources, and so forth? 

• How does it fit in with local staff needs? 

• What outcomes are expected? 

• How long will it take? 

• How will it be evaluated? 

• Do the initial training ideas make sense? 

• Are we on schedule? 

• Are arrangements being made? 

FULL SCALE IMPLEMENTATION PHASE 

• What is the impact of individual training sessions? 

• What is the overall impact of the program on ''^ff 
members? 

• Are staff implementing new ideas with students, 
community members, etc. 

• Are new activities starting? 

• What are staff perceptions of the inservice training 
programs? 

• What kinds of before/after effects can be observed? 

• Are staff members using suggested resources? 
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FACILITATOR OUTLINE 



NOTES 



• Are staff members developing new materials? 



• How would you show others that your staff develop- 
ment program has really had an impact on individuals 
involved? 



6. Indicate that ideas generated by the groups will be noted 
on the chalkboard. Also note that about 20 mi' ^s 
should be allotted for generation of the two r. . 9s. 



As an option, the facilitator might move around the room I 

working with groups and if desired noting interesting ideas in j 

the matrix outline on the chalkboard as they are produced. 1 

Another option, if time is short, would be to conduct the 1 
activity as a large group laboratory. The large group could j 
generate ideas for one matrix (preferably for the full-scale i 
implementation phase) and the facilitator could jot the ideas j 
on a sample matrix sketched on the chalkboard. 1 
1 
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ACTIVITY TITLE: 


DURATION: 


Module Summary 


5 minutes 


FACILITATOR OUTLINE 


NOTES 


A. Review Module Ideas 




Show the transparency "Evaluation as a Monster/' p. XI 1-37. 






1 . Note that we started by describing and dealing with 
evaluation as a kind of monster or ogre that generally 
brinos to mind neostive nerrentinns /Rpmpmhpr thp 
quiz.) 




2. We then nrocGpHed to Hpornhp* 




# Evslustinn cnmPthinn \Ayp Hn In ov/cr\/r1o\f l!fn 
^ wvaiuanVi/ii ao Owlildlllliy WC UU III CV^i yUdy IIIC* 




* Define evaluation 




* Show the role of evaluation in relation shin to thp 
Rural America Series. 

— Evaluation in the olannina ohase to pmnhasizp thp 
need to evaluate that phase. 

- Evaluation of the overall program impact. 




• Note that planning for evaluation should be started 
early in the process of program planning. 




3. Evaluation concepts and ideas were then extensively 
explored. 




• Users of evaluative information. 




• Information needs of users. 




• Strategies for collecting that information. 




• Ways of reporting that information. 




• Phases of programs-planning, tryout, and 
implementation 




• Sample evaluation considerations for the pianning and 
implementation phases. 
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FACILITATOR OUTLINE 


NOTES 


4. Lastly "hands-on" opportunities were provided. 
Hopefully this enabled participants to 

• Gain a feel for the evaluative process. 

• Develop a sense of competence in their ability to 
conceptualize and deal with concrete evaluation 
procedures. 

• Direct participants (time permitting) to Exhibits B1 to 
BIO in the evaluation handbook. These pages contain 
many ideas for program evaluation, especially during 
the program planning phase. 


pp. 58-79, Deciding Via 
Evaluation 


Show the transparency, "Evaluation as a Friend/' p. XI 1-38. 


5. End by noting that hopefully evaluation is no longer 
a monster and 

Plan to evaluate 
Don't procastinate 
Delete the negative 
Expand the positive. 

Don't forget 
It's a certain bet 
You have the will 
And the skill. 
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LISTING OF PARTICIPANT MATERIALS 



Material Page(s) 

Evaluation Quiz for the Rural America Series XI 1-30 

Evaluation of Phase (Blank Matrix) XI 1-31 

Staff Development Ideas XI 1-32 

Module-Specific Evaluation Questionnaire XI 1-33 
(To be used only if this module is 
used independently of others) 
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EVALUATION QUIZ 
FOR THE RURAL AMERICA SERIES 



Directions: Each of the following questions was selected to measure your understandings and 
perceptions of evaluation. Read each question carefully and then circle the letter of the statement 
that represents your answer to the question. After you have completed the quiz, the module 
leader will give you further instructions. 



Evaluation Multiple Choice Quiz 



1. Which of the following words or phrases best describes your first thought when you hear 
the word evaluation? 

a. Hooray 

b. O.K., already 

c. Ugh! 

d. Yuck! 

2. How would you describe evaluation? 

a. Testing program. 

b. Something which 111 probably have to do. 

c. Someone rating or ranking you. 

d. All of the above. 

e. A valuable program and planning tool. 

3. How would you characterize your abilities to conduct a program evaluation? 

a. It's difficult for me to do. 

b. I need a consultant. 

c. You've got to be kidding. 

d. Sometimes easy/sometimes difficult. 

4. In your judgment, which animal or beast represents evaluation. 

a. Teddy bear (warm and cuddly) 

b. Kitty cai (fun but at times hard to understand) 

c. Fully scented skunk (self-explanatory) 

d. Ogre (self-explanatory) 
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EVALUATION OF PHASE 



"^\^Evaluation 

Level ^^^"^-^ 


Who Is (Are) 
the Decision 
Makers? 


What Questions or 
Ccncerns Do 
They Have? 


How Can That 
Information Be 
Collected/Obtained? 


Hovir Should That 
Information Be 
Reported? 













STAFF DEVELOPMENT IDEAS 



Content Ideas for Career Guidance In Rural and Small Schools 

Forming active planning committees 
Needs assessment procedures 
Resource assessment procedures 
Developing behavioral objectives 
Planning procedures 
Curriculum ideas for the classroom 
Programs requiring community involvement 
Career resource centers 
Placement programs 

Community relations and program publicity 
Programs for parents 
Program evaluation 
Career guidance resources 

Cooperative agreements with business-industry-labor 
Rural community attitudes toward career development 
Research on career development 



Inservlce Techniques for Career Guidance In Rural and Small Schools 

Small group discussions Independent study 

Role playing Brainstorming 

Lecture Demonstration 

Large group discussions Panel discussions 

Large group laboratories Field visits 

Simulation Small group work-sessions 
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RURAL AMERICA SERIES 
Module Questionnaire: Evaluation 



NAME (Optional) TITLE 

I NST ITUTI ON 

ADDRESS TELEPHONE 

DIRECTIONS: Directions for responding to this brief questionnaire are given with each specific 

question. Your responses will be used to improve the module structure and format. 



1. 



On the left indicate the degree to which the module helped you in gainirg the understandings 
and skills listed below. On the right provide brief comments on how the module could be im- 
proved for those sections marked "NONE" or "SOME." 



Degree o 


' Growt 


h 


NONE 


SOME 


MUCH 


VERY 
MUCH 



Understandings/Skills 



Comments 



1 2 3 4 Increased understanding of the 

nature of evaluative thought 
processes. 



1 2 3 4 Increased understanding of the 

differences between and similiari- 
ties of program planning and 
program implementation evalua- 
tion. 



2 3 4 Developed skills in the planning 
of evaluation activities. 



To what extent were the materials, processes, and organizational aspects of the module suc- 
cessfully used in the presentation and delivery of the module. For those materials, processes 
or organizational aspects that you marked as "unsuccessful" or "slightly successful" provide 
brief comments as to how they might be improved. 
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Success 



Materials/Processes 



Comments 
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3 
3 

3 
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3 



3 
3 



4 
4 

4 
4 
4 



4 
4 



Materials 

Transparencies 

Rural America Series handout 

Processes 

Lecture Presentations 

Small Group Work Sessions 

Question and Answer Sessions 

Organizational Aspects 

Module Organization in Terms 
of the Logical Flow of Ideas 

Important Concepts Reinforced 

The Mix of Activities Helpful 
in Maintaining Interest 



3. Indicate those aspects of the module that you liked most and those that you liked least. 
Liked Most Comments 



Liked Least 



Comments 
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4. SUGGESTIONS: 



Please provide any suggestions or comments that you have for improving 
the workshop, workshop materials, etc. 



Thanks for your help. 
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LISTING OF TRANSPARENCY MASTERS 



Transparency p^g^^^, 

The Rural America Series XI 1-36 

Evaluation as a Monster XI 1-37 

Evaluation as a Friend XI 1-38 
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THE RURAL AMERICA SERIES (HANDBOOKS) 



PROBLEM IDENTIFICATION 



Support 
Information 



State of the Art 



Planning 
Process 



Planning and 
Implementation 



PROBLEM RESOLUTION 



Career Guidance & 
Counseling Strategies 



Career Counseling 



Support 
Functions 



Staff 

Development 



Supplemental 
Areas 



Facilitator's 
Guide 



Model 



Career Guidance 
Resources 



Needs 
Assessment 



Behavioral 
Objectives 



Resource 
I Assessment 



Desk Reference 



Individualized 
Counseling and 
Placement 



Transitional 
Career Placement 



Community 
Relations and 
Involvement 



Community 
Perspectives 



Cooperative 
Agreements 



Case Studies 



Evaluation 



Career Guidance 
Practices 
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SUPPLRMENTS TO THE RURAL.AMERICA SERIES {1978) 
Increasing Guidance Effectiveness Through School-Community Cooperation 
Fronr^ Idea to Action: Career Guidance Tians of Rural and Small Schools 
Facilitator's Gjide to Staff Training for the Rura/ America Series ' ' - 

RURAL AMERICA ^E^IES (rnQrll) ; 

Career Guidance Program Support Information Docyments . ^ 

State of the Art Review . \ 

Life Role Development Model 
Career Guidance Resources 

Career Guidance Program Process Handbooks 

Planning and Inipjementation 

Career Development Needs Assessment ^ 
Behavioral Objectives 
Resource Assessment 
Deciding Via Evaluation 

Career Guidance and Counseling for Groups and Individuals Handbooks 

Career Counseling in the Rural School 

Desk Reference: Facilitating Career Counseling and Placement 

An Individualized Approach to Career Counseling and Career Placement 

Transitional Career Placement in the Rural School 

Career Guidance Practices 

Career Guidance Program Support Functions Handbooks 

Staff Development 

Communit/ Relptions and Involvement 

Rural Community Perspectives Toward Career Development 



For ordering information (individual and quantity prices) and/or in-service training technical 
assistant, contact: 

The N-^tional Center for Research in Vocational Education 

Natl ' - Center Publications 

The w^iiio State University 

1960 Kenny Road 

Columbus, Ohio 43210 

(614) 486-3655 
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